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Abstract 

^a/«v 0/ n.t.ork ccu.ections can ofi.n W o larr,e impact on «.e performance of autri^t^ a,j.UcoU<^ 
For e^mplc. </oc„m«n. tmr^fer applicatious as FTP, GopUr .ni the WoHi WU. Wck s^er ir^o.c4 r^por^e 
« of n^t^ork congestion. For these apportions, the ^ument transfer time is 4ir^Uy relaUi to tte 

«v..a Me y..tn Of the connection. A.aUa^U ^ on t^ things: S) the .n^eriyin, 

^m cUnt to server. .McH is iimite, Me bottleneck .ink; an, 2) the amount of other traffic competing for links on the 

c^c^at^. Network uUli^tion coW. then U use. as a ^is for selection „ „ 

thus p W.„, re,«ce^ response time. Such a dynamic server selection scheme 6. espci^Uty important in a motUe 

computing environment in M the set of available servers is fr^ently changiny. 

In onier to provide these measurements at the application level. v,e intr^uce tv^ tools: bprobe, v.hich provides an 
Of the unccnyeste, han^u^m of a path; an, cp..be, u,hich yives an estimate of the current congestion along a 
path. These measures may 6e W .„ comtination to p^,e the application ^ an estimate of avaUahU ian<lu.id.k 
betv.een server anj client thereby enabling appUcation^level congestion avoidance. 

In this paper v,e discuss the design and implementation of our probe tooU, specifically illustrating the techni^es used 
to achieve «ccun«:, and robustness. We present validation studies for both tooU v.hich demonstrate their reliability in the 
face of actual Internet condiHons; and u.e give results of a survey of available bandu.idth to a random set of WWW servers 
as a sample application of our probe technique. We conclude v,iih descripHons of other applications of our meosur^ent 
tools, several of which are currently under development. 

1 Introduction 

For m^y appUcations. the quaUty of network connections can have a large impact on performance. One im- 
portant characteristic of a networlc connection U the bandwidth available to cUents using that connection. In 
particular, for document transfer applications (e.,. FTP. Gopher and the World Wide Web) higher available 
bandwidth implies fester document transfer time. AvailaWe bandwidth depends on two things: 1) the miderlying 
capacity of the path between cUent and server which is limited by the slowest (or boWcneck) link, and 2) the 
presence of compeUng traffic (congestion). 
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One technique to minimize the time between document request and document arrival begins with an eval- 
uation of the quality of a set of connections to candidate servers. The application may then choose to retrieve 
the document from the server with the highest quality connection. In this context, the quality of the network 
connection U determined by an estimate of network utilization which can be used to assess potential transfer 
time. Utilization, in turn, depends on both the bandwidth of the path (limited by the bottleneck link) and the 
presence of other traffic competing for the path. 

However, neither of these two useful pieces of information are readily available to appUcations. In order to 
discover tliis information we developed two tools: bprobe, which measures the uncongested bandwidth of the 
bottleneck link of a connection; and cprobe, which estimates the current congestion along the bottleneck link 
of the paUi. When applications are provided with measurements of the current network state, application-level 
congestion avoidance becomes possible. By application-level congestion avoidance we mean that applications 
can use estimates of traffic volume to actively avoid paths that are currently heavily loaded. 

The measurement of bottleneck Unk speed involves sending a series of ICMP echo packets from source to 
destination and measuring the inter-arrival times between successive packets. Under certain condiUons analysis 
of the distribution of the inter-arrival times leads to a straightforward, yet reliable, calculation of the bottleneck 
link speed of the connecUon. Bprvbe is specifically designed to make the necesaary conditions occur and performs 
the analysis required to make a robust estimate of bottleneck link speed. 

The method of measurement of competing traffic used by cprvbe also rdies on echo packets. An estimate of 
congestion can be made based on the elapsed time between the return of the first and last of a series of packets 
and the amount of data sent by the probe tool. Cprobe is designed to make a reliable estimate of competing 
traffic under real network conditions. 

In the following sections we motivate and describe bprobe and cprobe. We present the theory behind the tools 
and explain the obstacles to making reliable measurements in practice. We then explain how the the probe tools 
were designed to overcome these obstacles. We also describe the validation process in which we compared the 
measuremeuts made by our tools to known link capacities on local, regional and wide-area networks. We present 
the results of a survey of WWW servers and discuss the implications of our findings for replication and server 
selection. We conclude with a discussion of ongoing and future work. 

1.1 Related Work 

la (2). the author gives a model for packet travel through links and routers along a connection in the Internet. 
The bandwidth of the bottleneck link is studied by comparing the round-trip times (rtt) of adjacent packets 
from a sequence sent at regular intervals. The rtfs of successive packets are plotted against each other {r«,- vs 
rtt.-,). The inverse of the slope of a line fitted to this data gives the bandwidth and the Intercept gives the 
latency. The line fitting Ls done by hand and a very large number of packets are needed lo make manual fitting 
possible. An expanded version of that paper gives preliminary suggestfons as to how this process can be used 
to determme the makeup of interfering traffic [1]. However, many obsUeles exist to the automatic appUcation 
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of this idea in real networks. Our work overcomes these obstacles and results in a robust procedure that can 

calculate the bottleneck link speed with minimal overhead. 

The author in [6], defines the notion of a pachct-pair, two back-to-back packets that travel from source to 

destination and result in two acknowledgment pad^ets returning. The calculation of bandwidth based on the 

returning ACKs is simUar to ours but in that work, a network of Rate-AUocating Servers (RAS) is assumed. 

With RAS each incoming stream of packets is served in turn with one packet processed from each queue in its 

turn if any packets are present. This in effect shares the packet processor equaUy among all incoming lines. 
Thus, the actual bandwidth available to the source can be accurately measured since the inter-arrival time of 

two packets from the same source will be increased by the processing time of one packet fh>m each competing 
input line in use. In contrast, we assume the conditions prevalent in the current Internet, which are much less 
tractable, and we require only that packets not be reordered in transit. 

IVeno [3| is a tool from Pittsburgh Supercomputer Center that emulates a TCP connection including slow- 
start and necessary retransmissions. They have set up a Web-based server that measures the path back to the 
callmg client. U.sing a scheme similar to traceroute, it sends UDP packets with increasing TTL along the path 
from the server to the invoking client. Then it reports the available bandwidth that TCP would find along each 
of the path prefixes to the destination. In order to get the full effect of TCP, it is recommended that the tool be 
run for at least 10 seconds of continuous traffic. In contrast, our cprobe tool measures the estimated available 
bandwidth without regard to any particular higher-level protocol while our hprobe tool measures the underlying 
bottleneck link speed. In addition, we find that our measurement process is typically faster than the 10 seconds 
necessary for treno to reliably measure tlie bandwidth available using TCP. 

2 Measuring Bandwidth and Congestion at the Application Level 

We use the term base ba^width of a comiection to mean the maximum transmission rate that could be achieved 
by the comiection in the absence of any competing traffic. This will be limited by the speed of the bottleneck 
link, so we also refer to this as the botUeneck link speed. However, packets from other connections may share 
one or more of the links aJong the conneclion we want to measure; this competing traffic lowers the bandwidth 
available to oi,r application. We use the term availabh bandundth to refer to the estimated transfer rate available 
to the appUcation at any instant. In other words, the portion of base bandwidth which is not used by competing 
traffic. We define the uMization of a connection as the ratio of available bandwidth to base bandwidth, that is, 
the percentage of the base bandwidth which should be avaUable to the application. 

If a measure of current utilization is avaUable. appUcations can make informed resource allocation deci- 
sions, fbr the specific problem of WWW server selection, knowledge of current network utilizaUon allows better 
prediction of information transfer time from the candidate servers. 

With this objective in mind, we set out to develop tools to measure the current utilizaUon of a connection. 
These measurements are necessarUy done from a distance and in a unknown envin^mnent. Therefore, we refer 
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Source 




Fi^e 1: Path from Source to Target. Initially, the Unk speeds are unknown, 
to our measurement process as prt^bing and the tools as probes. Our design goals for the probes were that they: 

• should work at the appUcation level; i.e. by usable by any user program; 

• should have a minimal unpact on both the network and the server we are probing; 

• should not rely on cooperation from the network or the server; 

• should be robust and perform properly under varying conditions; and 

• should be as accurate as possible while providing timely information. 

The following sections describes the design and validation of BPROBE and CPROBE. 

2.1 Bprobe: Measuring Base Bandwidth 
2.1.1 Measurement Theory 

In the following discussion we assume a network like the Internet. In such a network, a path between any two 
hosts in the network is made up of one or more link,. Between each set of Unks along the path is a router which 
examines the destination address of each packet and forwards it along the appropriate outgoing link. Our main 
requirement of the network is that packets are not frequently reordered in transit. We also assume that the path 
is stable, by which we mean that the path packets take at any instant will not change for the next few seconds, 
at least. For the Internet, rouUng table updates are infrequent enough that this is a reasonable assumption. 
We further assume that the bottleneck in both directions is the same link, although this assumption could be 
relajCRtl in a different design. 

Recall tliat the output of bprobe is a measurement of the base bandwidtli of a connection: the speed of the 
bottleneck link. The situation is as presented m Figure 1. In order to compute utilization, it would be most 
useful to know the minimum speed among the Unks along the path from the Source to the Target. The method 
used by BPROBE is to send a sequence of ICMP ECHO packets from the source to the target and measure the 
inter-arrival times of the returning packets. 

Figure 2 illustrates the journey of a pair of packets along the round-trip path fr«m source to target and back. 
When the packets depart from the Source host, the inter-departure gap is measured as (d, - di). As the packets 
aow through intermediate routers, the inter-packet gap may change as the packets Bow over Unks of various 
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Figure 2: Flow of bprobe packets from Source to Target and back. The inter-packet gap is stretched by the 
bottleneck link. The inter-arrival time measured at the Source can be used to calculate the bottleneck link speed. 

capacities. In general, the size of the gap varies inversely with the capacity of the link. Figure 2 shows that the 
gap between the packets is stretched when the packets flow through the bottleneck link. On the return trip the 
packets flow again over the bottleneck link and the gap is unchanged. When the packets return to the Source, 
the inter-arrival time (aj - o,) reflects the speed of the bottleneck link. 

The essential idea behind the probe, then, is this: if two packets can be caused to travel txigether such that 
they are queued as a pair at the bottleneck link, with no packets intervening between them, then the inter-packet 
spacing win be proportional to the processing time required for the bottleneck router to process the second packet 
of the pair. This well-known effect is illustrated in the familiar diagram shown m Figure 3 (adapted from Van 
Jacobson [5]). In addition, Bolot describes the basic effect in (1. 2] and Keshav has used a similar method in 
networks of Rate-Allocating servers [6]. 

The goal of the bprobe tool is to create this condition and to use it to make reHable measurements. In other 
words, if packets from the probe tool alone are queued at the bottleneck link, then the inter-arrival Ume of those 
pairs that were queued can be used at the endpoint of the path to estimate the base bandwidth of the bottleneck 
link. Under ideal conditions, the receiver can use this informatMn to measure the bottleneck link speed as follows: 
the trailing edge of the first of a pair of packets marks the time when the router started processing the second 
packet of the pair and the trailing edge of the second packet records the Ume when the router finished processing 
that packet. Given a packet of si,* V. and the inter-arrival time (or gap), we can estimate the base bandwidth 
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Figure 3: Illustration of packet How through a bottleneck Unk. 
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Figure 4: Ideal case: probe packets alone queued at router, 
of the bottleneck link, Bm^i as follows 

jPw, bytes/second = ^ ^^^^ . 

gap seconds 

This is iUustrated in Figure 4 which shows the situation when probe packets alone are queued at a router. 
2.1.2 Obstacles to Measuring Base Bandwidth in Practice 

However, in contrast to the the network architecture assumed in (6), in our experimental envirunment (the current 
Internet) this ideal behavior is not easUy achieved. There are se»^al problems that arise in practice: 

• Queuing foilure: Probe packets may not queue at the bottleneck link. 

• Competing traffic: Competing traffic along the path may intervene between probe packets. 
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• Probe packet drops: Packets sent by the probe may be dropped. 

• Downstream congestion: Congestion at routers downstream from the bottleneck may invalidate the 
results. 

Each of these problems can be the cause of spurious estimates of base bandwidth. The major obstacle to 
implementation, then, is this: given a set of inter-arriva] time measurements, how can the probe tool decide 
which will result in valid bandwidth estimates? The challenge was to start from the intuitive idea captured 
in Figure 3 and design an accurate, robust and low-impact tool to measure bandwidth. We now present our 
solutions to these problems. 

2.1.3 Solutions to Implementation Problems 

Both BPROBE and CPROBE are built on top of the ICMP ECHO mechanism. Because of our use of ICMP ECHO, 
a client can send packets to a ho»t and receive replies without installLug new software at the remote site, which 
affords wide utility of our tools. 

There are two principal techniques with which we attack these problems. First, the use of multiple packets 
of varying sizes; second, the tool uses a careful filtering process which discards inaccurate measurements. 

Queuing failure: The first problem to overcome is that the client sending the probe packets may not happen 
to send data fast enough to cause queuing at the bottleneck router. In order to ensure such queuiag, we send a 
number of packets and we use packets of varying sizes. Larger packets will naturally take more processing time 
at routers and increase the possibiHty of queuing. Therefore, we want to use as large a padcet as can negotiate 
the round-trip path. The upper limit on packet size will vary from path to path, however, so we use packets of 
varying size. 

Currently, the probe runs in distinct phases with each phase using packets of successively larger sizes. The 
first phase sends a number of packets (currently 10) of the smallest size that will be used (currently 124 bytes). 
The next phase uses even larger packets and this process continues until we reach the majdmum packet size 
which our test client can send (approximately 8000 bytes). In this way, we adapt to the maximum feasible size 
for each connection. 

Competing traffic: The second problem is the interference of other traffic sharing a link along the path. In 
the ideal case, no non-probe packets intervene. One or more intervening packets will cause an increase in the 
inter-arrival time and therefore an underestimate of the bottleneck link speed. This is illustrated in Figure 5 
which shows a non-probe packet queued between two probe packets. The resulting inter-arrival time measures 
the processing time for the V bytes of packet P2 as well as the unknown number of bytes contained in the 
intervening packet. Therefore, the estimated link speed will be an underestimate. 

The solution to this problem is two-fold. By sending a large number of packets, we increase the likelihood 
that some pairs will not be disrupted by competing packets. Even when many pairs are disrupted, the number 

7 



PACE 37/61 • RCVD AT 12/20/2008 8:28:10 PM [Eastern Standard TimeJ • SVR:USPTO-EFXRP-6/4e • DNIS:2738300 • 0810:9149821973 " DURATION (mm-ss): 22-30 



Dec 20 06 08:40p 



RhlSE V.DOUGHERTY 



9149621973 



p. 38 



l» [» 1^ 




rXIEIEIOE] ED 



rnmoDoiz] h h 
t 1 



d a time to process 
P bytBs of P2 4 
7 7 cblier byces 

P/dslta 



FiRure 5: Competing packets intervfining between probe packets cause an underestimate of link speed. 

Of intervening b>te5 wUl often varjr from pair to pair. This results in differing bandwidth estimates which may 
then be filtered to determine the correct estimate. We explain the filtering process below. 

As a further measure, we increase the packet size by alternating factors of 150% and 250%. This ensures 
that no two packet sizes are integer multiples of each other. If we simply doubled the packet size, the bandwidth 
estimated when one packet of size x intervenes between probe packets of size » would be the same as the 
bandwidth estimated when two packets of size x intervene between probe packets of size 2y. As will be clear 
from the discussion of our filtering method, this could easily result in an erroneous final estimate. The use of 
alternoting increments diminishes the probability of a bad estimate. 

Probe packet drops: The third problem that must be addressed is dropped probe packets. Some packets 
are simply lost, but large packets are more likely to cause buffer overflow and be dropped. Large packets will 
be fragmented and this also increases the likelihood of dropped packets due to fragment loss. (Aside from this 
effect, fragmentation of large packets has not been a problem for our tools.) We avoid this problem by sending 
packets of varying sizes. Sending many packets of each size also serves to make the probe tolerant of a few packet 
losses regardless of the cause. 

Downstream congestion: The fourth problem occurs when queuing occurs on the return trip, after passing 
through the bottleneck hnk. That is. even if all goes weU from cUent to the server and back through the bottleneck 
link, congestion at intermediate servers downstream from the bottleneck can invaUdate an estimate. Consider 
a pair of packets whose inter-packet gap was properly set by the bottleneck link. If these packets subsequently 
get queued, the inter-packet gap will be reset, thus measuring the wrong link. Since this subsequent queuing is 
unlikely, we can alleviate this problem during filtering. Some of the pairs may, in fact, measure the wrong link 
but if enough of the pairs make it bade without further queuing, the erroneous esUmates wiU be filtered out. 
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Figure 6: 5 BProbe experimcQts to local 56Kbps hosts 

The Filtering Proce.^: The most difficult problem that mu3t be addressed by the probe is identification of 
those estimates that should be used to determine the base bandwidth from those that should be discarded due 
to lack of queuing, subsequent queuing or interference from competing traffic. Each phi^ae of 10 probe packets 
results In at most 9 inter-arrival measurements. Thus, at most 7 sets of at most 9 measurements cew:h are the 
input to the altering process. Given the packet size (and some inteUigent guesses as to protocol headers and link 
level headers and trailers) each pair of packets generaies one raw estimate of the bandwidth oB^^ming queuing 
as defined before. 

For example, Figure 6 shows the raw estimates for 5 invocations of the probe tool. All of the data points 
belonging to one invocation are shown by plotting a numeral representing the invocation number at each data 
point. The abscissa of a daU point is the bandwidth estimate, and the ordinate increases with packet size. Thus, 
aU the measurements for the first invocation occur on the bottom 5 line segments of the graph, all those for the 
second on the next higher 5 lines and so on. Within each invocation, packet size is increasing from bottom to 
top. Notice the clustering of estimate values as packet size increases; this shows that, in general, larger packet 
sizes yield more consistent results. 

The multi-phase variable-packet-size design of the probe results in 1) correlation among correct estimates 
and 2) lack of correUtioa among incorrect estimates. Fbr example, suppose aU intervening packets are of size x, 
and the probe packet sizes for two probe phases are pX and p2 with pi p2. The estimates produced by the 
packet sequences pi - ar - pi and j,2 - x - p2 will both underestimate the baae bandwidth, but the Important 
characteristic is that the two estimates wUl differ. On the other hand, sequences like pi - pi and p2 -p2 wiU 
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produce agreeing corroct estimates. Ot>>er incorrect estimates such as produced by the sequence pi - x - a; - x -pi 
will be even less correlated with the correct estimates and also uncorrclated with other. It is these properties of 
the data, which are evident In Figure 6, that we seek to exploit using non-linear filtering. 

We have tested two filtering methods both of which rely on computing an ^'error interval" around each 
estimate, and subjecting these intervals to further set operations as explained below. The error interval can 
be expanded as necessary untU a satisfactory estimate of the speed of the bottleneck Unk is determined. The 
intersection filtering method finds overlaps between estimate intervals and computes the intersecUon, with the 
idea being that we want to find the estimate that occurs in all sets. Since we observed that larger packets provide 
better estimates we start intersecting using the estimates firom the largest packets and iteratively intersect with 
estimates derived from successively smaller packets. The union filtering method combines overlapping intervals 
using set union, and selects an interval only if enough sets contribute to it. Both methods produce one interval 
as the final result and the midpoint of this interval is returned as the final estimate. 

The tvo methods arc illustrated in Figure 7, in which two sets of estimates arc combined using either Union 
or Intersection. In the top of the figure are histograms representing the two sets. The horizontal axis gives 
intervals of estimated bottleneck bandwidth (increasing to the right) and the vert.ical axis gives the number 
of measurements m each interval. The lower portion of the figiu-e represents the histograms resulting from 
combination using the Union or Intersection operations. For the union result, the height of the histogram bar, 

and the number of sets which contribute to it. is shown under each interval using the notation: For 
example, the fourth interval has a total of 7 measurements which origuiate in 2 sets. In this case the only interval 
that has members from more than one set is the (7,2) interval and the midpoint of this interval would be the 
resulting estimate of bottleneck bandwidth. 

In our experience the union approach shows less dispersion of estimates than intersection and we have adopted 
it as the filtering method currently used by DPROBE. 

2.1.4 Validation of BPROBE 

To validate the base bandwidth probe tool, we performed three sets of experiments. We tested bproBE between 
a fixed host and hosts on our local network, on our regional network, and on the wide-area network. For each 
of the networks we used avaQable maps of fink capacities to determine the bottleneck link speed for each of the 
target servers. Therefore, we were able to annpare the results of the tool against the known bottleueck link 
speeds. 

Local vaUdation: First, we tested the probe to a set of 16 hosts on our local network, 9 of which were 
connected by Ethernet and 7 of which were connected over 56Kbps lines. Each minute over a period of 4 days we 
ran the probe tool and recorded the bandwidth estimate. The results of these tests are sunmiarized In Figure 8 
and Figure 9 which presents histograms of bp rode bandwidth estimates for the two classes. In both figures, 
the histogram bins are labeled with botUeneck Unk speeds in bits-per-second. Figure 8 shows the histogram 
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Figure 8: Histogram of B PROBE results: local 56Kbps hosts 

of the estimated bottleneck Uak speeds for the 7 56Kbps hosts; Figure 9 shows the histogram of the estiniated 
bottleneck link speeds for the 9 Ethernet (10Mbps) hosts. Both figures show the majority of the estimates closely 
clustered about the expected values, even though the expected values of the two host sets differ by two orders 
of magnitude. 

Regional validation: The next set of tests was performed within NearNet, the regional network to which our 

site belongs. 

eer« we were able to add a third category of bottleneck capacity: Tl. For the regional network we have 
sets of hosts with bottleneck link capacities of 56Kbps, 1.544Kbps (Tl) and 10Mbps (Ethernet). Once again the 
measurements were made periodicaUy over a few days and the results are presented as histograms of bottleneck 
link speed estimates. 

Figure 10 shows the histogram of bottleneck bandwidth estimates for the hosts known to have a 56Kbps 
bottleneck. The histogram of bandwidth estimates for 681 probes of the 6 hosts shows a very tight grouping 
of estimates. Although there is a coniriatent underestimate, we still find 73% of the estimates within 5% of the 
theoretical capacity. 87% of the estimates within 10% of the theoretical capacity and 92% of the estimates within 
^% of the theoretical capacity 

For the hosts with a Tl bottleneck, Figure 11 gives the histogram of 1156 probes of the 9 hosts. The picture 
is a little less clear for this case as there appear to be several minor peaks in the data. However, we stiU find 
most of the estimates dose to the rated capacity In particular, we find: 23% of the estimates within 5% of the 
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Figure 10: matogram of bprobe results: Nearnet 56Kbps hosts 
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Figure 12: Histogram of BPROBE results: Nearnet Ethernet hosts 
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Figure 13: Histogram of BPROBE results: SURANET 56Kbps hosts 

rated capacity, 52% of the estimates within 10% of the rated capacity and 75% of the estimates within 20% of 
rated capacity. 

Finally, Figure 12 shows the histogram for regional hosts connected by Ethernet. The 397 probes of tlie 3 
Ethernet hosts resulted in another tight grouping, with fiactiles comparable to the results for the SOKbps hosts. 
For the 10Mbps hosts we find: 68% of the estimates within 5% of the 10Mbps capacity. 86% of the estimates 
within 10% of the 10Mbps capacity and 92% of the estimates within 20% of the 10Mbps capacity. 

Wide-area validation: The hnai validation test was done using a set of hosts on SURANET, a regional 
network in the Baltimor^Washington, D.C. metropolitan area. Hosts on this regional net are approximately 16 
hops away from our test client including 7 MCI backbone hops. Once again, we were able to obtain capacity 
maps which let us independently verify the capacity of the bottleneck link to each of the selected hosts. 

Figure 13 shows the histogram of bandwidth estimates for the 61 1 probes of the 4 56Kbps hosts. Once again, 
the histogram shows a tight grouping of estimates near the known bottleneck speed. 

Figure 14 shows the Wstogram of bandwidth estimates for the 459 probes of the 3 Tl hosts. In this case, as in 
the regional case, we find minor peaks around the main peak at the expected value. Wc suggest an explanation 
in the Discussion section below. 

Figure 15 shows the histogram of bandwidth estimates for the 475 probes of the 3 Ethernet hosts. Here we 
find a lair amount of serious underestimates, but the bulk of the data is stUI clustered around the known value. 
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Figure 14: Histogram of bprobe results: SURANET Tl hosts 
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Figure 15: Histogram of bprode results: SURANET Ethernet nodes 
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S ummar y and Discussion Relative error values for the three sets of hosts are given in T^ble 1 , which presents 
a more quantitative view of the validation results. The columns show percentages of error relative to the rated 
capacity of the bottleneck link; each row gives measurements for a particular subset of hosts. Each entry in the 
table gives the percentage of the bandwidth estimates that fall within a given relative percentage of the rated 
capacity. For example, 96% of the bandwidth estimates of local 56Kbps hosts were within ±10% of 56Kbps. As is 
evident from Figure 8 there is a systematic imderestimatton error due to unaccounted for link-level encapsulation 
overhead. In practice, this should not be a significant problem. For example, this could be addressed using a 
table of commonly available capacities to round the bandwidth estimates to the nearest likely value. 

The results for Tl connections at the regional and wide-area levels were surprising. In contrast to the single 
large peaks found in the histograms for the other link capacities, for the two Tl host sets we find midtiple peaks 
in the histograms (Figures 11 &14). An immediate suggestion was that the peaks correspond to individual hosts 
and represent underlying hardware differences. However, this is not the case. Bottleneck link speed estimates 
for any particular host were found in many of the peaks. 

Out hypothesis is that consistent cross-traffic consisting of small packets of nearly equal size could cause 
this effect- Such traffic might be generated by telnet sessions, for example. This competing traffic would result 
in underestimation of bottleneck bandwidth as explained in section 2.1.3. If the source of the cross-traffic was 
regular enough, many of the resulting underestimates produced by BPROBE would agree and thus defeat our 
filtering process. The resulting estimate of bottleneck link speed for the host would then be an underestimate. 
Nevertheless, as measured by the fractile statistics, the estimates in these cases are still fairly accurate. 





Relative Error 


Network 


Rated Capacity 


No. Hosts 


±5% 


±10% 


±20% 


Local 


56Kbps 


7 


55% 


96% 


99% 


10Mbps 


9 


80% 


97% 


100% 


Regional 


56Kbps 


6 


73% 


87% 


92% 


L54Mbps 


9 


23% 


52% 


75% 


lOMbps 




68% 


86% 


92% 


Wide-area 


56Kbps 


4 


84% 


93% 


97% 


1.54Mbps 


3 


12% 


54% 


82% 


lOMbps 


3 


31% 


53% 


63% 



Table 1: Measurements of BP robe accuracy for local, regional and wide-area host sets. 

The results of these three sets of experiments convince us of the accuracy of the probe tool. For three very 
different classes of hosts we were able to accurately measure the base bandwidth of links using bprobe. As we 
move from local to regional and then wide-area hosts, the tool shows only a minor loss in estimate accuracy. 
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Considering now our stated goals, we find that bprobe does provide accirate base bandwidth estimates in 
spite of the difficult environment it measures; it operates without expUcit cooperation from thft network or servers; 
and it provides reliable estimates without excessive time overhead. At the current time, a reasonable concern is 
the probe's impact on the network. The ciirrent implementation consumes too much network bandwidth. Two 
approaches are under development: First, we believe we can achieve nearly the same accuracy and reUabUity 
with fewer packets; and second, it is anticipated that the baae bandwidth estimate of a path will be cached, so 
that redundant measurements of a path will be unnecessary. 

2.1.5 Survey of Bottlenecks to WWW Servers 

As a ftirther demonstration of the use of bprobe wc surveyed a set of 5825 randomly selected WWW servers 
to get an idea of the distribution of base bandwidth of connections to servers on the WWW. The servers were 
choeen uniformly firom a list obtained from (7]. 

Previously, we performed a similar survey primarily concerned with the distribution of hops and latency to 
WWW Servers [4]. During the latest survey we gathered these Statistics in addition to bandwidth estimates. 
For the set of 5825 servers. Figure IG presents the distribution of hops (measured by tracenmte) and Figure 17 
gives the distribution of latencies (measured by piru,). As discussed in [4] the striking diSerence between the two 
distributions has impUcaUons for rephca placement and server selection. In particular, the lack of correlation 
between the two measure suggests that hops is a bad predictor of latency, and thus a bad metric for distance 
in an internetwork. What is needed is a dynamic measurement of current network conditions. This observation 
motivated us to design bprobe and cprobe. 

Figure 18 shows the histogram of bottleneck link speed estimates from this survey of WWW server. Notice 
the distinct peak at 10Mbps. There ai* no higher estimates because the survey was run from a machine on 
a 10Mbps local network. Since bprobe measures the bottleneck link, readings higher than this should not be 
expected. Another concentration of estimates appears around 1 Mbps. 

Inspecting the data, we find that of the 6825 servers surveyed, 2586 have bottleneck bandwidth estimates 
greater than 5Mbps. In other words, about 44% of the servers were reachable at Ethernet speeds. However, 
nearly 5095 of the servers surveyed had a bottleneck link speed less than half of Ethernet speed and nearly 40% 
of them exhibit a bottleneck less than 20% of Ethernet speed. 

In Figure 19 we restrirt the histogram to estimates smaller than 200Kbps. This range includes servers with a 
low-speed modem as the botUeneck as weU as other low-capacity connections. There are 659 (about 10%) of the 
servers with bottlenecks in this range. There is a clear peak at 56Kbps, a value which we have established as a 
common bottleneck speed in our validation experiments. Another peak appears near the ISDN rate of 128Kbps. 

Figure 20 gives estimates in the range 200Kbps to 2Mbps, a region which includes Tl comiections, frattional 
Tl and other values. This region includes 2351. or about 40% of the servers surveyed. 

What are the implications of this distribution for replication and server selection? If we assume a uniform 
distribution of copies of popular documents over this set of servers, it becomes clear that measurement of the 
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Figure 16: Ilistograxu of hops to 5825 WWW Serves 
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Figure 17: Histogram of latencies to 5825 WWW Servers 
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Figure 18: WWW survey results: liiatogram of bottleneck link speeds to 5825 WWW 
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Figure 19: WWW survey results: subset (10%) of servers with bottleneck link speeds less than 200Kbps 
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5825 WWW Servers [Zoom \n on (200.000 to 2.000.000 bps)] 
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Figure 20: WWW survey results: subset (40%) of servers with bottleneck link speeds between 200Kbps and 
2Mbps 

bottleneck link speed is of prime importance when a choosing among replicas. For this distribution, choosing 
a server at random gives a better than even chance of getting a slow link. Clearly, a choice based on avaUable 
bandwidth should give better performance. More precise evaluation of the aznount of hnprovement to be gained 
by dynamic server selection policies are the focus of our current work. 

2.2 Cprobe: Measuring Available Bandwidth 

In the previous section we described our bandwidth probing tool which gives a fairly rcUable estimate of the 
bottleneck Unk speed for a path between two hosts on the Internet. We now consider estimating available 
bandwidth. 

To measure available bandwidth, we developed a tool called cprvbe. Cprobe's technique is straightforwaH: 
by bouncing a short stream of echo packets off of the target server and recording the time between the receipt 
of the first packet and the receipt of the last packet, we can measure the presence of competing traffic on 
the bottleneck link. Dividing the number of bytes sent by this Ume yields a measure of available bandwidth, 
Ba.au. that represents the actual throug».put achieved by the probe bytes. As long as we can send packets at a 
higher rate than the bottleneck link speed (which we can measure using bprobe) this effect should occur. Any 
additional time lag between the first packet and the last one represents delay due to intervening non-probe bytes 
(competing traffic). 

The utilization of the bottleneck link can then be computed as the ratio of available bandwidth measured by 
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CPROBE to the bottleneck link speed as estimated by bprobe: 



In practice, we discovered a compUcation during testing of cprobe. OccasionaUy. the sending host would 
be momentarily delayed due to operating system effects, delaying the returning flow of packets. This resulted 
in an unusually long delay between one pair of packets which then caused an overeBtimate of competing traffic, 
since the delay was wrongly attributed entirely to traffic. In addition, scheduling efi^ects on the receiving side can' 
cause the inter-packet time to be unrealisticaUy short. To eliminate these two kinds of erroneous readings from 
om- d*ta we discard the highest and lowest inter-arrival measurements when calculating We have found 

that thi3 improves Uie accuracy of the measurements significantly. In order to tolerate packet drops and possible 
re-ordering of packets (which invalidate inter-arrival time raeas,ixements), we use the results of four separate 
8-packet streams when calculating the available bandwidth. 

As in the case of bprobe. care must be taken to ensure that the cprobe's results are valid. We validated the 
individual inter-arrival measurements using a packet tracing tool running on a local Ethernet. The experimental 
set-up consisted of the probe client and the probe target; a hoet running the packet trace tool; and other hosts 
between which FTP sessions were run to provide a hacVgromid load. While varying the background load we ran 
several repetitions of the probe tool. We tlien compared the probe's „.easurements of packet inter^val times 
w,th the log of the packet traces. The probe's measurements of elapsed time for parket transfer* wer^ generaUy 
accurate to within ±10% relative to the packet tra« measurements. We then compared CPROBF/s estimate of 
available bandwidth with that derived bom the packet trace log. Using the log. we calculated the time difference 
between the first and last reply, and divided by the amount of data sent. dupUcating the calculation done by 
CPROBE. The measurements are quite accurate as can be seen from the f^actile results in T^ble 2. Three quarters 
of the measurements are within 5% of the actual value and all are with 25% of the actual value. Of course, these 
results are limited to the local-area network where we can control the cross-traflic and measure all probe and 
uon-probe packets. Beyond the local network vaUdation becomes very difficult. 



Relative error FVactile 


Percentage of meaaurements within FVactile 


5% 


74% 


10% 


81% 


15% 


88% 


20% 


92% 


25% 


100% 



Table 2; FtactiJe quantities for cprobe's available bandwidth estimates. 



Evaluating cprobe against our design goals we End that it operates without relying on support from servers 
or the network; that it's impact on the network is not too great and that the measurement of available bandwidth 
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is quite accurate. An open question is the predictive abUity ofths measurements of available bandwidth provided 
by CPROBB. Can the measurements be used to reliably predict the available bandwidth in the future? If so, how 
far into the future and with what degree of accuracy? 

3 Future Directions and Conclusions 

We have plans to extend this work in several directions. 

Our priniary interest is using bprobe and cPnoBB in addition to ping to gather current infonnation about 
the state of the network and u.ing this information as input to dynamic server selection algorithms We plan 
to evaluate the contribution of each of these measurement techniques and combinations of them to determine 
which are of use in selecting a server dynamically. 

CurrenUy. we are building a venuon of the Mosaic WWW browser which uses the BPROBE tool to inform the 
user of the relative expected tranrfer speeds of links on a Web page. The hypertext anchor is color-coded with a 
measure of the current network state. For in.stance. fa.,ter expected transfers are coded green whUe slower ones 
axe red. 

Another browser extension we are considering is support for multiple URL. per link in a document. This 
hst of alternate servers can be used as input to dynamic server selection by the browser. This represents a step 
along the way to integrated support for replication in which the browser would have to obtain a list of repHcas. 
Such a mechanism might be particularly attractive to heavily loaded sites. 

We also plan to pad«ge the probe tools as a daemon that can be placed strategicaUy throughout a networic 
to allow probing of conditions in remote parts of the network. This would also allow measurement of a Unk as a 
part of several paths providing confinnation of probe estimates. 

fa conclusion, we have introduced and validated two tools usefi:! for measuring network conditions at the 
application level: BPROBE. which uses ECHO packets to measure the bottleneck link speed of paths between 
hosts in the Internet; and cpkobe, which measures the presence of competing traffic on the bottleneck link. We 
have presented vaUdaUon results for each tool showing that accurate and reUable measurements of bottleneck 
link speed can be made under real network conditions. As an example application of our tools we presented a 
study in which we used bprobe to measure the base baiidwidth to a set of WWW servers. The results of this 
survey Ulustrate the potential for performance improvements based on dyrmmic measurement of current network 
conditions such as provided by bprobe and cprobe. 
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Abstract— The Internet is changing rapidly. One of the 

^ * ^'^'''^ fo'- higher 

quality of service. A corollary of th« nood for higher quality 
Of service is the need for accurate and extensive m^ure- 
naent, mcludinR the need to mAAjture bandwidth. 

Currently the most accurate bandwidth measurement 
techniques are to directly measure the fastest rate that traf- 
fic can be sent thrt>ugh a network. Wlde^scale deployment 
of these 'heavy-weight' bandwidth tests can av«rwholm the 
^^u""^ 7-1^ '.^ Accurate measurement of hand! 

width IS difficult if simple large data volume techniques are 
not used but there is some current research in this area. 
We desCTibe existing techniques that attempt to estimate 
the bandwidth of both links and paths while attempti^ to 
use as small a quantity of data ai« possible. These tech- 
niques must operate from only the end points of a connec- 
tion, and must not require specialist software be deployed 
Mnd^'^r f ^f^oTtc, We explain the theory be- 
hmd two popu ar bandwidth estimation techniques, single 
packet and pocket pair techniques. The discussion includes 
the of the techniques and the problems each encoun- 

ters. We also describe some of the Improvements that have 
been suggested. 

Finally we explain the differences between current tech- 
niques and the goals we are trying to achieve and give an 
early proposal describing the areas we plan to develop ft^ 

KcywoTxia — Bandwidth estimation, active network mea- 
surements, bottleneck bandwidth, network performance 
measiu-emcnt. 



I. Introduction 

With the increase in use of the Internet, more people 
are finding themselves dependent on it. Just as happened 
with the telephone system early last century, business and 
people are finding that the requirement of Internet for com- 
munication and gathering of information is something they 
cannot operate v,-ithout. Increasingly more and more busi- 
ness models are based solely around the Internet. 

However with this growth of dependency and use of the 
Internet, more and more demands are being placed on the 
performance of the network. Users require that consistent 
monitoring of the performance is carried out, in order to 
both detect faults quickly and predict and provision for the 
growth of the network. 

Measuring the Internet is difficult, some of the reasons 
for this are described below. Not all ISF's are forthcoming 
about details of the loading and performance of their net- 
work. Even with the support of the ISP, the complexity 
of the network means that normally multiple providers are 
mvolved in the end-to-end connection between hosts. This 
situation makes tlie monitoring of end-to-end performance 
by any one ISP nearly impossible. 



The inability for users to be confident in the performance 
m the network is causing a demand for tools to enable users 
to asses the performance without assistance. These tools 
need to quickly and easily measure the end-to-end perfor- 
mance of the network, while not placing anymore load on 
the network than is absolutely necessary. Extra load may 
restrict the times that tlie measurement may be made, and 
depending on the charging system, could create large extra 
traffic cliarges. 

One possible way to meet this need would be the de- 
ployment of special software or hardware on each router in 
the network. A soluUon sudi as this, however, is just not 
practical. The cost, time and security problems with this 
outweigh the gains from this type of instrumentation. The 
cost of this solution is involve in the man hours spent 
upgrading software on aU of the routers in the network, 
the cliarges for this software by the vendor, and the price 
to upgrade older routers that are unable to run this soft- 
ware. This sort of upgrade is also not going to be instan- 
taneous. The time required for upgrading the software on 
every router in the entire network would be huge. This 
would leave a substantial time where there are inconsisten- 
cies in the network when it may be possible to measure 
some of the paths, and not others. 

An alternative approach is to use end-to-end software run 
on the end hosts. This allows the measurement to be run 
at the users discretion and allows for simple deployment. 
However this approach requires the software to infer the 
characteristics of the links involved without being able to 
directly measure each link individually. 

This paper explains and discusses the current research 
and techniques used towards solving this demand. Also 
discussed is our motivation to become involved in this area 
and an early proposal of the approach we plan to take. 

The rest of this paper is organised as follows. Sec- 
tion II discusses and explains the midtiple definitions dif- 
ferent groups apply to common terms in this area. For 
clarity, the definition of the terms we will use are given. 
Section m presents our interest in this area. Sections IV 
through to VII describe the techniques ahready proposed 
and some example implementations. Section VIII explains 
the techniques we wiU be investigating to meet our objec- 
tives. We conclude in section DC. 

II. What is ^bandwidth* ? 

Like most specialist research areas, bandwidth measure- 
ment and estimation has many specific terms that need to 
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Figure 1. bandwidth and latency- 



be explained. Unfortunately many of these terms are used 
differently by different authors. This section clarifies the 
main terms, and defines how wc will uae them in this paper. 

We begin with the names for the components of a net- 
work. Hosts are the end points from which a packet either 
on^natea from or is destined to. A router is a machine 
with two or more network connections that forwards pack- 
ets from one connection to another that will get the packet 
closer to its destination. A link refers to a single connec- 
tion between routers or routers and hosts. A path is the 
collection of links, joined by routers, that carries the pack- 
ets from the source to the destination host. Two paths are 
different if any Intermediate router is different- 
Figure 1 shows a link between two routers. Lirik latency 
is the time it takes from the time the first byte of a packet 
is placed on the medium until the time that the first byte 
is taken from the medium. This delay is caused by the rate 
at wluch signals are propagated in the Unk (e.g. electrons 
in a cable) and distance of the medium. 

The link bandwidth is the rate at which bits can be in- 
serted into the medium. The faster the bandwidth the more 
bits can be placed on the medium in a given time frame. 

An important thing to note about computer network 
routers is that they are normally store-aud-forward routers. 
This means that every byte of the packet must be received 
from the Unk and placed into a buffer in the router before 
the router will stArt to send it out on the destination link. 
If packets arrive at a router faster than they can be sent 
out the appropriate output port a packet queue will form 
for tliis port. The queue discipline used is almost always a 
FIFO queue. 

In the next sections we go into more detail about the 
different variations of the terms latency and bandwidth 

A. Latencies 

There are many separate latencies in a network system. 
Each of these values are used throughout this paper so we 
describe them all here. 

The transmission delay is the time it takes a packet to 
be placed on the medium. Tliis time is proportional to the 
packet size and the bandwidth of a link. It is the time from 
the time the first byte is placed on the network until the 
time the last hytc has been sent. 

The imTismission time is considered to be the combina- 
tion of link latency and transmission delay. The transmis- 
sion time is the time between the first byte being placed 



on the medium and the last byte being taken off. This is 
tlie sum of the link latency and the transjuission delay. 

Tlie path latency is the sura of all of the individual 
transmission times as well as the queueing time inside the 
routers. This is the time that it takes from the sender is- 
suing the packet until the destination receiving it. Path 
latency is often referred to as the one-way delay . 

Path latency is hard to measure as it requires a dis- 
tributed synchronised clock to timestamp the time the 
packet was scut and the time the packet was received. The 
clocks need to be synchronised as the sending time will 
be timestamped from the senders clock and the reception 
timestamp from the destination machines clock. 

A more common measurement is the rottnrf trip time 
(RTT) latency. This is the sum of the path latency in 
the forward and reverse dfrecUons, and can be measured 
easUy by timing the sending of a packet, have the desUna- 
tion machine respond to the packet immediately and the 
original sender tiraestamp the return of this packet. 

Unfortunately as the forward and reverse paths and path 
latencies may differ, the RTT latency cannot differentiate 
between the forward and reverse delay. 

B. Bandwidths 

As with latency, there are many variants of the term 
bandwidth. This section wiU discuss these. 

Unlike latencies, link bandwidths do not sum to result in 
the path bandwidth. The path bandwidth is defined by the 
minimum of the link bandwidths, as this ig the fastest any 
traffic can make it through the path. The path bandwidth 
is alao known as the path capacity. 

The bandwidth of a path is shared by the traffic imder 
consideration and other traffic. This reduces the amount 
of bandwidth available to the hosts. Tliis other traffic is 
referred to as cvoss traffic. 

Available bandwidth is the amount of bandwidth "left 
over" after the cross traffic. The link with the lowest avaU- 
able bandwidth will not necessary be the link with the low- 
est capacity. 

Dovrolis et al, (Ij refer to the link that restricts the paths 
capacity as the narrow link and the link that restricts the 
avaUable bandwidth the tight link . 

III. Motivation 
A large amount of money and time is currently being 
spent implementing high speed, next generation networks. 
These networks are being constructed in-order to support 
the large growth in the Internet, as well as enabling more 
high bandwidth services to run over the netvvork to more 
people. There is an increasing demand to know if the per- 
formance obtained from these networks is what is expected 
from them. The performance of a network is a complicated 
Issue with many variables effecting different traffic in differ- 
ent ways. While poor values for some performance metrics 
may only have a strong effect on the performance of a small 
number of applications, there are a few metrics which are 
ahnost universally significant for all types of traffic. Band- 
width and latency are the two most commonly quoted of 



PACE 56/61 ' RCVD AT 12/20/2006 8:28:10 PM [Eastern Standard Time) • SVR:USPTO-EFXRF-6/46 • DN18:2738300 " 0810:0140621073 • DURATION <mm-ss): 22-30 



Dec 20 OB 08:47p 



RMNE V.DOUGHERTY 



9149G21973 



p. 57 



these performance metrics. 

RTT measurements are the simplest of the metrics to 
measure. They are easy to make suitably accurate with- 
out any specialist hardware, and place a light load on the 
network being measured. Because of this there are several 
research projects that take re?gular frequent RTT measure^ 
raents between a large and diverse range of hosts on the 
lntemet([2][3](4][5)). 

A. AMP 

One such project is NLANR's [6] Active Measurement 
Project (AMP)[2]. As of February 2001 AMP consisted of 
approximately 120 machines, connected to high speed re- 
search networks (such as the vBNS and AbUene) placed 
throughout the USA. Each machine takes reguiax RTT 
measurements to every other machine, resulting in mea- 
surements of approximately 14,000 paths. 

These results are coUected and made available on the 
Web in real-time, enabling network operators and users of 
the network to view, not only the current state of RTT 
performance for any path, but also the history of the RTT 
measurements back as far as six months. 

The AMP group wish to extend the information they 
provide to their users by including regular available band- 
width measurements for all paths. 

The concern is that if the measurements of available 
bandwidth themselves create a large quantity of traffic then 
this will effect not only the users of tlie network, but also 
the results of the bandwidth and RTT measurements from 
the other machines. 

IV. Existing Techniques 
There have been a number of techxiiques proposed in the 
area of bandwidth estimation. Most concentrate on mca- 
soring one of two values, either the individual link band- 
widtlis of a path, or the capacity of a path. In general these 
techniques can be classified into two groups. Single packet 
and packet pair techniques. The names refer to the number 
of packets that are used in a single probe. A measurement 
of a link or path will consist of multiple probes, in the 
case of some implementations [7] this can be in the order 
of 10MB of data (14400 individual probes) to measure a 
10 hop path. The foUowing sections will detail the theory 
of these techniques, improvements suggested and example 
implementations. 

V. Single Packet TECHNrquRs 
Single packet techniques concentrate on estimathig the 
individual Unk bandwidths as opposed to end-to-end prop- 
erties. These ter.liniques are based on the observation that 
slower links will take longer to transmit a packet than faster 
links. If it is known how k>ng a packet takes to cross each 
link, the bandwidth of that link can be calculated. 

Calculations must also take into account the latency, 
which varies for each link. As discussed in jn and fig- 
ure 1, latency is not dependent on the packet size or the 
bandwidth of that link, but the tune that the signal takes 
to travel down the path. The transmission time of a packet 




Packet Size (bytes) 

Figure 2. CaJculating the slope of this graph forma the basis of how 
single p>acket techniques estimate bandwidth. 



is determined by the packet size (P), the bandwidth of the 
link (6) plus a fixed latency (/) vahie. 



If the time and packet size are known equation 1 can 
be rearranged to ^ve the bandwidth. An latency is fixed 
for a particular link, latency can be considered as a fixed 
offset. When the transmission time for multiple, varied 
sized probe packets are taken, a graph such as figure 2 can 
be produced. Each probe packet is plotted using packet size 
vs transmission time. The bandwidth can be calculated 
from this graph by performing a linear regression to find 
the slope of the points, and the mverse of this value is the 
estimated bandwidth. 

There are addition problems when conducting these mea- 
surements in the real world. The issue of measuring the 
time it takes a packet to cross each link in the network 
path separately is the first hurdle. 

To avoid the need for special router instrumentation, sin- 
gle packet implera^tations take advantage of the IP time 
to live (TTL) field^ This field is decremented by one by 
each router the packet passes through. Once the TTL is 
decremented to zero, the packet is discarded and the router 
will send an ICMP TTL expired error message to the orig- 
mal packet source. By settmg the TTL to expire at the 
router at the end of the link to be measured, the sender 
can record the KTT of the packet to the end of the link 
and back by recording the sent time of the packet and the 
return time of the ICMP error message. 

An example of a measurement can be seen in figure 3. 
In this example, host A is measuring the path to host B. 
More specifically this figure depicts the measurement of the 
second Hnk of the path to B. For the sake of clarity we have 

^Thc term Time Tb Live is a historical oqc. OriginaUy the field 
referred to the time that a packet had before it cxpirod- Thi« field 
now refers to the number of hops (links) before the packet expires, 
and not a time vutue. 
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Figure 3. By using the IP TTL field, the host can make mcaaure- 
ments to each router without needing specialist software 



omitted to show either the measurement of the first ]mk, 
or any of the successive links after this one. 

The TTL for the measurement packet is set to 2 as it 
leaves the machine. The TTL is decremented to 1 in the 
first router, and 0 in the second router. This router then 
generates an ICMP error message to be returned to host A. 
The error message may or may not follow the same path 
to return to host A and for this reason is displayed as a 
dashed line. The effect of the return path differing fi-om 
the forward path wiU be discussed later in this section. 

Although the use of the TTL field allows measurements 
to be made from the end points without special software 
deployed in eadi router in the path, it also causes a num- 
ber of problems. First there is no way of considering any 
link (with the exception of the first link in the path) inde- 
pendently. The measurements reported for all but the first 
link must also include aU the effects of the previous links. 

This problem is solved by representing a link as a sura 
of aJl the results for the previous links, plus the component 
of this link. The latency for a single link was given in 
equation 1. 

Summing the links we derive: 



(2) 



This means that the latency of the link bdng measured 
can be calculated by subtracting the latency measured on 
the previous links. The bandwidth of the link is calculated 
by subtracting the slope of the previous links from the slope 
of tliis link and taking the inverse of this value. 

This process has the disadvantage that errors accumu- 
late over each Unks measurements. This problem results in 
a limit to the accuracy that can be expected for measure- 
ments of distant links in long hop count paths. 

However, this is only one of many more problems that 
single packet techniques encounter. Possibly the most sig- 
nificant of these problems is the effect of other traffic on 



the link (cross traffic). If a packet experiences delays, due 
to cross traffic on Uie link, then the estimation of time will 
be affected proportionally to the volume of this traffic. 

Jacobson [7| addresses this issue by assuming that cross 
traffic can only ever increase a delay seen by a packet. If 
enough packets are sent eventually one should get at least 
one through in the minimum time. These packets are said 
to have the shoHest observed round trip Hme (SORTT) and 
is discussed by Downey in [8]. The graph shown earUer 
in figure 2 could now be considered to be just plotting 
the SORTT values, and ignoring the delayed packets that 
would appear above each measurement. A graph showing 
all measurement packets is discussed later in the paper, 
and is shown in figure 6. 

More packets are required to discover the SORTT for 
more disUnt links. As the results from links further away 
from the source are combined to the effects of all of the 
previous links a packet must experience no queueing delay 
through every node, not just the node being measured. 

While single packet techniques have many difficulties 
there are a number of common problems in the area of net- 
work measurement that do not strongly affect the band- 
width estimate results. Asymmetric routing is one such 
issue. 

The term asymmetric routing refers to when the path too 
and from a node is different and, because of this, the delay 
in each direction can be different. This can cause prob- 
lems for many types of active and passive measurement 
analysis. In the case of single packet bandwidth estimar 
tion techniques asynmietric routing causes few problems. 
The reason for this is wlule the ICMP error packet that 
we need for timing may come back via a different path, 
the error packet is of fixed size for all sized measurement 
packets. Assuming the route doesn't change, the time it 
takes through the return path wUl, therdbre, be fixed for 
all packet sizes. However, we cannot distinguish between 
added packet delay due to congestion on the forward and 
reverse paths. This means tliat the packet must experience 
zero delay through all routers on both the forward and re- 
verse paths. This means that we need to send more packets 
to discover the SORTT. 

Asymmetric routing will however cause problems with 
the latency estimation described earlier. As the return path 
may change speed for different hops, the subtraction of pre- 
vious delay times will no longer hold true. This will result 
in an incorrect estimation of the latency added on by this 
Unk. An example of this is shown in figure 4. In this mea- 
surement, packets returned from router 3 travd over a low 
latency link when compared to the links that measurement 
2 has to travel over. Using the example latencies shown in 
the figure, the measurement for link two will be the com- 
bined latency of the paths, summing to 40ms in this case. 
The measurement for link 3 however will be carried out 
over 3 lOma paths, and back over 2 2m8 paths and one 
lOms path. This gives a sum of 44m5 RTT. The estimated 
RTT latency for this link will then be 44ms - 40ms, 4m.s. 
This compared to the correct value of the 10ms link {20ms 
RTT) latency. 
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Figure 4. Latency errors caused by asymmetric routing 



Another issue causing problems with the latency estima- 
tion is caused by the time taken by a router to generate the 
ICMP error message. ICMP can be used as a form of de- 
nial of service attack on a router. To reduce this risk many 
routers place a very low priority on the creation of ICMP 
packets to avoid overloading the routers CPU. This means 
that the latency observed will include this extra time that 
would not be seen by a packet traveling through a router. 
This will not affect the bandwidth estimation, however, as 
long as tlie router is consistent on the time introduced for 
all packet sizes. If the router handles packets of different 
sizes differently tliis would introduce errors into the slope 
calculation used to calculate bandwidth. 

The combination of these problems means that this tech- 
nique doesn't scale to higher hop counts. The amount of 
traffic required to accurately find the SORTT also reduces 
the usefulness of this method on busy paths. 

The noise introduced into the measurement by cross traf- 
fic, and the other problems discussed here, is often larger 
than the difference between the transmission of the small- 
est packet (40 bytes) and the largest packet (normally 1500 
bytes). Downey [8] discusses measurements where the dif- 
ference between the largest and smallest packet size mea- 
surements is approximately IbOfis wliile the largest outlier 
from the fitted slope is 175/43. The fixed latency offset for 
a good link would be d[ the order of 2m$, while this could 
climb cosily to 20mB for long haul links. Intercontinental 
links, such as New Zealand to the USA can even reach a 
RTT of 120m5 or greater. 

A, Implementations 

There have been a number of example implementations 
of single packet techniques. The most common implemen- 
tation is Jaoobson's pathchar [7]. Two indei>endent clones 
of Jacobson's patbchar tool are clink [0] and pchar [10]. 
Each of these use slightly different algorithms on how to de- 
cide when the SOKXT has been discovered, but each uses 
the same basic algorithm described. 




LI L2 L3 

Figure 5. Packet Pair meaaureraent through the limiting link. 

VI. Packet Pair Models 

Packet Pair models attempt to estimate the path ca- 
pacity not the link capacity disrx>vered by single packet 
techniques. These techniques have been in use since at 
least 1993, when Bolot [11] used them to estimate the path 
capacity between Prance and the USA. He was able to 
quite accurately measure the transatlantic capacity, which 
at that time was 128kbp3. 

Packet pair techniques are often referred to as packet dis- 
persion techniques. This name is perhaps more descriptive. 
A packet experiences a serialisation delay across each link 
due to the bandwidth of the Unk. Packet pair techniques 
send two identically sized packets back-to-back, and mea^ 
sure the difference in the time between the packets when 
they arrive at the destination. 

For simpUcity, we will initially ignore cross traffic, and 
discuss the effects it has later in this section. Figure 5, 
modified from [8], shows packet pair measurements dia- 
grammatically. Shown are 3 links of a path firom source 
S to destination D, and two packets, shown once in each 
link. Link LI and L3 have twice the capacity of L2; L2 is 
the capacity limiting link in the path. The first packet ar- 
rives at the router between LI and L2 and is forwarded out 
on L2 without queueing delay as there is no other traffic 
present. L2 has a lower speed than LI so the second packet 
arrives while the first packet is still being sent out and is 
queued. As soon as the first packet has been transmitted 
down L2 the second packet begins to be sent. As soon as 
the first packet is received by the router between L2 and L3 
the router can forward it out on L3. The first packet will 
have finished being sent before the second packet is fully 
received by the second router as L3 is faster than L2. Afl 
soon as the router does finish receiving the second packet 
it will forward it on. As the spacing can only be changed 
by a slower hnk, and we have defmed L2 to be the capacity 
limiting (slowest) hnk, the spacing will remidn the same 
through to the destination machine. 

The spacing is equal to the time that the router at the 
end of the limiting link spent receiving the second packet af- 
ter the first one was received. This because the first packet 
was sent as soon as it was fully received and then the router 
must wait until the second packet has fully arrived before 
it can be sent on. This value is actually equal to the trans- 
mission delay. The transmission delay (t) proportional to 
the packet size (P) and the capacity of the link (t). 



T^P/b 



(3) 



Note the difference between equation 1 and 3. Single 
packet techniques have to take into accoimt the latency 
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of the link in the calculations. Packet pair techniques do 
not need to estimate the latency of the link as it will be 
the same for both packets (in the absence of cross traffic), 
canceliug it out. 

Cross traffic efiFects are the most obvious and serious 
problem to affect packet pair measurements. If cross traffic 
delays the first pocket it will compress the spacing between 
the packets and the bandwidth estimation will be high. 
This is referred to as time or probe compression. If cross 
traffic arrives in the queue between the first and second 
packet the spacing will be expanded resulting in underes- 
timation of the bandwidth. 

All recent research into packet pair techniques has fo- 
cused on filtering out the compressed or extended mea- 
surements to provide the closest estimation to the capacity 
of the path. 

Lai et al. [12] and Carter et al. (13j both propose statis- 
tical methods to filtering the results to estimate the band- 
width. Both of these approaches assume that as the com- 
pression or extension values will be random, then the actual 
bandwidth should appear as the most conunon measure- 
ment. 

However current research from (1] suggests that this may 
not be the case, and that there may be other, more common 
values than the actual bandwidth. This result is the focus 
of current research into packet pair techniques. 

VII. Other Techniques 

Lai and Baker follow up (12] with a paper describing 
their design and implementation of a measurement tool 
[14], This technique, which they call Packet TaiigaUng, 
can be considered a combination of both single and packet 
p^r techniques. 

This technique aims to discover both link and path ca- 
pacities. The derivation of the formulas that form the basis 
of tailgating are discussed in detail in [14], and are outside 
of the scope of this paper. 

The project aimed to obtain the link capacity measure- 
ments, without some of the limiting factors that are present 
in single pack^ techniques. Issues such as asymmetric 
routing, routers delaying the return of ICMP error mes- 
sages, rate limiting or firewalling of ICMP packets and 
effects added on during the return path are some of the 
problems tailgating avoids. 

The basic measurement technique is as follows. If two 
packets are back to back, but unlike packet pair techniques, 
the first packet is much larger than the second, the second 
packet will keep "catching up" to the first packet. This is 
because the transmission delay of the first packet is larger 
than the second. This means that the smaller packet is 
ready to be sent instantly from the router once the first 
packet has been sent. If the larger first packet is set to 
expire (using the same TTL method discussed in |V) at 
the link being measured, the second smaller packet will be 
"released" firom the delays cause by being behind the bigger 
packet. 

The latency experienced by the smaller packet will 
change if the link that the larger packet la set to expire 
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Figure 6. Example data for single i>ax:ket naeasurement. 



changes. The link bandwidth is inferred by inspecting this 
change in latency. More details on this process are con- 
tained in [14]. 

VIII. Proposed Rrsf.arch 

The two major techniques described in §V and 5VI have 
both focused on finding the actual bandwidths of paths and 
links. These values are likely to be interesting to adminis- 
trators who wish to discover the topology of the network, 
or wish to investigate if the bandwidth allocated to them is 
what they are paying for. However these values are prob- 
ably not going to be so interesting to the end user. They 
simply want to know "How fast will my connection go ?" 

As stated in §HI-A AMP is a project that focuses on 
giving users of the network information about the current 
performance of the network. As such the AMP group is 
interested in, and the focus of our proposed research is on, 
estimating the available bandwidth and less on link or path 
capacities. 

This is the direction that this project is aims to investi- 
gate. Another major difference between our goals and the 
objectives of the current research is that current proposed 
techniques aim to give results on any network without any 
prior knowledge. This allows the tools to be general pur- 
pose. AMP wishes to deploy the tool on its current re- 
search infrastructure. Both a recent and distant histx>ry of 
the performance of the network as measured by the current 
tools wiU therefore be available. 

A primary goal is to investigate if these measurements 
can aid and support the bandwidth estimation. For exam- 
ple, the AMP measurements give information about the 
long term stability of the RTT measurements This stabil- 
ity of the paths may be useful in the bandwidth estimation 
process. 

It is also likely that the bandwidth estimation will be 
running consistently, allowing it to constantly change and 
adapt to the network as it changes, instead of the small 
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snapshot the current methods take whUe the program is 
collecting data. OnJy this data is used to calculate their 
resuJts. 

Both single and packet pair techniques use many of 
probes to find a snaal! number of probes that they are in- 
terested in, and discard the rest. Fbr example figure C is 
a graph of RTT vs packet size for a single link measure- 
ment taken by Downey in [8). AJI of the packets above the 
obvious SORTT baseline are discarded in the bandwidth 
estimation. We propose to investigate the relationship the 
distribution of the extra data has to the avaUable band- 
width of the path. Similar methods are proposed for packet 
pair techniques. 
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IX. Conclusion 

As the demand for performance on the Internet grows, so 
does the requirement for tools to accurately measure per- 
formance. This growing demand also means that solutions 
that place a large load on the network will not scale. This 
is creating a need for tools that can accurately estimate 
various types of baadwidths and do so without creating 
large volumes of traffic. 

In this paper we have presented a summary of the cur- 
renttechniques of bandwidth estimation. We have com- 
pared single packet techniques with packet pair models 
Also explained are the areas that these methods do not 
provide smtable accuracy. We have also presented existing 
suggestions on how to improve these techniques. 

FinaUy we have outlined the areas of research that this 
project will concentrate on and the possible differences our 
techniques wUl have to existing methods. 
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